Papers with network pruning
Distilling the Knowledge of Romanian BERTs Using Multiple Teachers (2022.lrec-1)
Copied to clipboard
Andrei-Marius Avram, Darius Catrina, Dumitru-Clementin Cercel, Mihai Dascalu, Traian Rebedea, Vasile Pais, Dan Tufis
| Challenge: | Existing approaches to train pre-trained language models focus on the English language, thus widening the gap when considering low-resource languages. |
| Approach: | They propose three versions of distilled BERT models for the Romanian language . they argue that the models offer performance comparable to their teachers . |
| Outcome: | The proposed models perform comparable to their teachers, while being twice as fast on a GPU and 35% smaller. |
Continual Learning for Task-oriented Dialogue System with Iterative Network Pruning, Expanding and Masking (2021.acl-short)
Copied to clipboard
| Challenge: | Existing methods to learn consecutive tasks without forgetting how to perform previously trained problems are lacking. |
| Approach: | They propose a continual learning method which preserves performance on previously encountered tasks while accelerating learning progress on subsequent tasks. |
| Outcome: | The proposed method preserves performance on previously encountered tasks while accelerating learning progress on subsequent tasks. |
SparseAdapter: An Easy Approach for Improving the Parameter-Efficiency of Adapters (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Pretrain-finetuned models are increasingly complex and require more parameters to match the performance of full fine-tuning. |
| Approach: | They propose an efficient Adapter Tuning technique that freezes pretrained language models and fine-tunes a few extra modules. |
| Outcome: | The proposed setting outperforms the standard Adapter Tuning by 80% . the proposed setting is easy to use and has a high sparse ratio . |
Specializing Pre-trained Language Models for Better Relational Reasoning via Network Pruning (2022.findings-naacl)
Copied to clipboard
| Challenge: | Pretrained masked language models inherit a considerable amount of relational knowledge from the source corpora. |
| Approach: | They propose to specialize pretrained masked language models into relational models from the perspective of network pruning. |
| Outcome: | The proposed model can represent grounded commonsense relations at non-trivial sparsity while being generalizable . the proposed model improves on a wealth of NLP tasks, but we know little about how much knowledge it imparts . |
Do Localization Methods Actually Localize Memorized Data in LLMs? A Tale of Two Benchmarks (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing studies on the ability of localization methods to pinpoint LLM components for memorized data are lacking. |
| Approach: | They propose to use a subset of LLM weights to evaluate localization methods . they propose to measure how much dropping out identified neurons deletes a memorized sequence. |
| Outcome: | The proposed methods show promising localization ability, despite differences in their evaluations. |
Finding Skill Neurons in Pre-trained Transformer-based Language Models (2022.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained language models have demonstrated superior performance on various natural language processing tasks. |
| Approach: | They find that after prompt tuning, some neurons encode task-specific skills . they also show that skill neurons are most likely generated in pre-training . |
| Outcome: | The neurons are highly predictive of task labels after prompt tuning for specific tasks. |
Low-Rank Prune-And-Factorize for Language Model Compression (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing methods to reduce parameter redundancy in pre-processed language models fail to retain satisfactory performance under moderate to high compression rates. |
| Approach: | They propose to use network pruning to extract low-rank sparsity pattern desirable to matrix factorization. |
| Outcome: | The proposed method has a superior compression-performance trade-off compared to existing methods. |